Physical movement simulation on a display device

ABSTRACT

A computer-implemented method includes: displaying, by a computer device, video content on a display of the computer device; determining, by the computer device, a moving object in the video content; and controlling, by the computer device, a plurality of microbots to move, on the display, over locations of the object in the video content.

BACKGROUND

The present invention relates generally to computer displays and, moreparticularly, to systems and methods for simulating physical movement ona computer display device.

Visual impairment, also known as vision impairment or vision loss, is adecreased ability to see. There are a number of approaches related tocomputer displays that assist people who suffer from visual impairment.

SUMMARY

In a first aspect of the invention, there is a computer-implementedmethod including: displaying, by a computer device, video content on adisplay of the computer device; determining, by the computer device, amoving object in the video content; and controlling, by the computerdevice, a plurality of microbots to move, on the display, over locationsof the object in the video content.

In another aspect of the invention, there is a computer program productincluding a computer readable storage medium having program instructionsembodied therewith. The program instructions are executable by acomputing device to cause the computer device to: display video contenton a display of the computer device; determine a moving object in thevideo content; and control a plurality of microbots to move, on thedisplay, over locations of the object in the video content.

In another aspect of the invention, there is system including aprocessor, a computer readable memory, and a computer readable storagemedium. The system includes: program instructions to display videocontent on a display of the computer device; program instructions todetermine a moving object in the video content; and program instructionsto control a plurality of microbots to move, on the display, overlocations of the object in the video content such that the plurality ofmicrobots move in synchronous motion with the object as the videocontent is displayed on the display. The program instructions are storedon the computer readable storage medium for execution by the processorvia the computer readable memory.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention is described in the detailed description whichfollows, in reference to the noted plurality of drawings by way ofnon-limiting examples of exemplary embodiments of the present invention.

FIG. 1 depicts a computer infrastructure according to an embodiment ofthe present invention.

FIG. 2 shows a block diagram of an exemplary system in accordance withaspects of the invention.

FIG. 3 shows an exemplary microbot in accordance with aspects of theinvention.

FIGS. 4A-C show an exemplary implementation in accordance with aspectsof the invention.

FIGS. 5 and 6 show exemplary implementations of a microbot on a displayin accordance with aspects of the invention.

FIG. 7 shows a flowchart of an exemplary method in accordance withaspects of the invention.

DETAILED DESCRIPTION

The present invention relates generally to computer displays and, moreparticularly, to systems and methods for simulating physical movement ona computer display device. According to aspects of the invention, thereis a system and method for physically displaying movement in a visualdisplay including: receiving video content, a portion of which includesa moving image; and replacing the moving image with a plurality of tinymovable robots that simulate the moving image. In embodiments, therobots display the video content.

Many visually impaired individuals are not completely blind and,instead, have some degree of vision. Such visually impaired individualsmay still be able to see some of the content displayed on a computerdisplay, but may not be able to see the movement of an object shown inthe content. To address this problem, aspects of the invention controlthe movement of plural robots on a surface of a computer display tosimulate the movement of an object in the content being shown on thedisplay. The physical movement of the robots in synchronous motion withthe content being shown on the display assists a visually impaired userin receiving the content.

Aspects of the invention provide a technical solution to the technicalproblem of visually impaired individuals having trouble using computerdisplays. In embodiments, the technical solution includes controllingthe movement of specialized machines in the form of microbots to move insynchronous motion with an object that is part of video content beingshown on a computer display. In some embodiments, the technical solutionalso includes using computer vision based object detection to determinea location of the object in the video content. In some embodiments, thetechnical solution further includes utilizing computer-basedself-assembly techniques amongst the microbots, where computer-basedself-assembly is rooted in computer technology.

In accordance with a particular embodiment, software installed on acomputer device identifies dynamic properties of digital objects(referred to herein as objects) that are moving in video content that isbeing displayed by a computer display of the computer device. Thedynamic properties may include one or more of: movement direction,movement speed, and type of movement. Based on the identifying thedynamic properties, the software causes a number of very small robots(e.g., microbots) to arrange over the object included in the videocontent, where the number and location of microbots are controlled toreplicate the same size and shape as the object. In embodiments, thecluster of microbots simulates the exact dynamic properties of themoving object and is synchronized with the moving object. Inembodiments, the movement pattern of each participating microbot iscalculated dynamically based on the movement propriety of the movingobject so that the combination of plural microbots simulates themovement of the object.

In embodiments, each of the microbots is covered with a display layer.In this manner, an array of these small display layers moves with theobject when the plurality of microbots moves with the object. In thisembodiment, portions of the digital content being displayed on thecomputer display are also shown on the respective display layers of themicrobots. In this manner, a user can visualize and/or feel the physicalmovement of the object with synchronized dynamic properties of theobject.

In additional embodiments, the arrangement and dynamic properties of themicrobots are controlled dynamically based on a level of zoom selectionand external touch force applied on the microbots. In aspects, when auser touches one or more of the microbots in the plurality of microbots,the system detects the touch pressure applied by the user and controlsone or more of the microbots to compensate for the applied touchpressure, e.g., by speeding up, applying more motive force, etc. In thismanner, appropriate power is generated by the one or more of themicrobots to attempt to maintain (or regain) the synchronization betweenthe microbots and the movement of the object in the video content.

In additional embodiments, the system includes control options (e.g.,through a service provider module) to flag objects in a video forsynchronization. In this manner, the system is configured to determineand eliminate some movements from the synchronization described hereon.In aspects, the system also limits the movement of the microbots usingsafety controls based on at least one of: boundary; dimension; size;color; location; and relative distance to surrounding objects.

In additional embodiments, the system includes a self-learning moduleconfigured to train the system based on user interaction andconfiguration pattern, feedback, and trend analysis. In thisembodiments, the system may also include a complementing scene analysistool configured to avoid synchronizing similar objects movement in othersimilar videos.

The present invention may be a system, a method, and/or a computerprogram product at any possible technical detail level of integration.The computer program product may include a computer readable storagemedium (or media) having computer readable program instructions thereonfor causing a processor to carry out aspects of the present invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, configuration data for integrated circuitry, oreither source code or object code written in any combination of one ormore programming languages, including an object oriented programminglanguage such as Smalltalk, C++, or the like, and procedural programminglanguages, such as the “C” programming language or similar programminglanguages. The computer readable program instructions may executeentirely on the user's computer, partly on the user's computer, as astand-alone software package, partly on the user's computer and partlyon a remote computer or entirely on the remote computer or server. Inthe latter scenario, the remote computer may be connected to the user'scomputer through any type of network, including a local area network(LAN) or a wide area network (WAN), or the connection may be made to anexternal computer (for example, through the Internet using an InternetService Provider). In some embodiments, electronic circuitry including,for example, programmable logic circuitry, field-programmable gatearrays (FPGA), or programmable logic arrays (PLA) may execute thecomputer readable program instructions by utilizing state information ofthe computer readable program instructions to personalize the electroniccircuitry, in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the blocks may occur out of theorder noted in the Figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

Referring now to FIG. 1, a schematic of an example of a computerinfrastructure is shown. Computer infrastructure 10 is only one exampleof a suitable computer infrastructure and is not intended to suggest anylimitation as to the scope of use or functionality of embodiments of theinvention described herein. Regardless, computer infrastructure 10 iscapable of being implemented and/or performing any of the functionalityset forth hereinabove.

In computer infrastructure 10 there is a computer system 12, which isoperational with numerous other general purpose or special purposecomputing system environments or configurations. Examples of well-knowncomputing systems, environments, and/or configurations that may besuitable for use with computer system 12 include, but are not limitedto, personal computer systems, server computer systems, thin clients,thick clients, hand-held or laptop devices, multiprocessor systems,microprocessor-based systems, set top boxes, programmable consumerelectronics, network PCs, minicomputer systems, mainframe computersystems, and distributed cloud computing environments that include anyof the above systems or devices, and the like.

Computer system 12 may be described in the general context of computersystem executable instructions, such as program modules, being executedby a computer system. Generally, program modules may include routines,programs, objects, components, logic, data structures, and so on thatperform particular tasks or implement particular abstract data types.Computer system 12 may be practiced in distributed cloud computingenvironments where tasks are performed by remote processing devices thatare linked through a communications network. In a distributed cloudcomputing environment, program modules may be located in both local andremote computer system storage media including memory storage devices.

As shown in FIG. 1, computer system 12 in computer infrastructure 10 isshown in the form of a general-purpose computing device. The componentsof computer system 12 may include, but are not limited to, one or moreprocessors or processing units 16, a system memory 28, and a bus 18 thatcouples various system components including system memory 28 toprocessor 16.

Bus 18 represents one or more of any of several types of bus structures,including a memory bus or memory controller, a peripheral bus, anaccelerated graphics port, and a processor or local bus using any of avariety of bus architectures. By way of example, and not limitation,such architectures include Industry Standard Architecture (ISA) bus,Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, VideoElectronics Standards Association (VESA) local bus, and PeripheralComponent Interconnects (PCI) bus.

Computer system 12 typically includes a variety of computer systemreadable media. Such media may be any available media that is accessibleby computer system 12, and it includes both volatile and non-volatilemedia, removable and non-removable media.

System memory 28 can include computer system readable media in the formof volatile memory, such as random access memory (RAM) 30 and/or cachememory 32. Computer system 12 may further include otherremovable/non-removable, volatile/non-volatile computer system storagemedia. By way of example only, storage system 34 can be provided forreading from and writing to a non-removable, non-volatile magnetic media(not shown and typically called a “hard drive”). Although not shown, amagnetic disk drive for reading from and writing to a removable,non-volatile magnetic disk (e.g., a “floppy disk”), and an optical diskdrive for reading from or writing to a removable, non-volatile opticaldisk such as a CD-ROM, DVD-ROM or other optical media can be provided.In such instances, each can be connected to bus 18 by one or more datamedia interfaces. As will be further depicted and described below,memory 28 may include at least one program product having a set (e.g.,at least one) of program modules that are configured to carry out thefunctions of embodiments of the invention.

Program/utility 40, having a set (at least one) of program modules 42,may be stored in memory 28 by way of example, and not limitation, aswell as an operating system, one or more application programs, otherprogram modules, and program data. Each of the operating system, one ormore application programs, other program modules, and program data orsome combination thereof, may include an implementation of a networkingenvironment. Program modules 42 generally carry out the functions and/ormethodologies of embodiments of the invention as described herein.

Computer system 12 may also communicate with one or more externaldevices 14 such as a keyboard, a pointing device, a display 24, etc.;one or more devices that enable a user to interact with computer system12; and/or any devices (e.g., network card, modem, etc.) that enablecomputer system 12 to communicate with one or more other computingdevices. Such communication can occur via Input/Output (I/O) interfaces22. Still yet, computer system 12 can communicate with one or morenetworks such as a local area network (LAN), a general wide area network(WAN), and/or a public network (e.g., the Internet) via network adapter20. As depicted, network adapter 20 communicates with the othercomponents of computer system 12 via bus 18. It should be understoodthat although not shown, other hardware and/or software components couldbe used in conjunction with computer system 12. Examples, include, butare not limited to: microcode, device drivers, redundant processingunits, external disk drive arrays, RAID systems, tape drives, and dataarchival storage systems, etc.

FIG. 2 shows a block diagram of an exemplary system in accordance withaspects of the invention. The system includes a computer device 100 anda plurality of microbots 200 a-n. The computer device 100 comprises oneor more elements of the computer system 12 of FIG. 1, such as aprocessor, bus, memory, and network adapter. In embodiments, thecomputer device 100 comprises a display 105, an analysis module 110, acontrol module 115, a communication system 120, and a learning module125. The display 105 is a video display such as an LCD (liquid crystaldisplay) or an OLED (organic light emitting diode) display, for example.

The analysis module 110, control module 115, and learning module 125 areeach program modules such as program module 42 of FIG. 1, and areconfigured to perform one or more of the processes describes herein. Inembodiments, the computer device 100 may include additional or fewermodules than those shown in FIG. 2. In embodiments, separate modules maybe integrated into a single module. Additionally, or alternatively, asingle module may be implemented as multiple modules.

According to aspects of the invention, the analysis module 110 isconfigured to analyze video content that is displayed on the display 105to determine a location of an object within the video content. Accordingto aspects of the invention, the control module 115 is configured togenerate control signals that are transmitted to the microbots 200 a-nby the communication system 120. The control signals cause the microbots200 a-n to move to the locations determined by the analysis module 110.In embodiments, the analysis module 110 and the control module 115 areconfigured to analyze the video content and control the microbots 200a-n in real time such that the microbots 200 a-n are caused to move insynchronous motion with the object in the video content.

In embodiments, the video content displayed by the display 105 isreceived from one or more external content sources 130 via a network140. The video content may comprise but is not limited to streamingvideo data. The network 140 may comprise a communication network orcombination of communication networks including one or more of a LAN,WAN, and the Internet. Alternatively to being received via the network140, the video content may be stored and accessed locally in permanentor removable memory of the computer device 100.

FIGS. 4A-C depict an exemplary use case that illustrates aspects of theinvention. As shown in FIG. 4A, the video content shown on the display105 includes a video of a windmill with blades 117 of the windmillturning in the direction D. In accordance with aspects of the invention,the analysis module 110 determines that the blades 117 are an objectthat is moving in the video content, determines a location of the blades117 in each frame of the video content, and passes the locationinformation to the control module 115. The control module 115 causes themicrobots 200 a-n to move to the determined the location of the blades,such that the microbots 200 a-n cover the image of the blades 117 on thedisplay 105 as shown in FIG. 4B. Continuing this example, the analysismodule 110 and the control module 115 perform this determination andcontrol continuously in real time with respect to the video contentshown on the display 105, such that the microbots 200 a-n are controlledto move on the surface of the display 105 in synchronous motion with theblades 117 of the windmill. In this manner, as the video content showsthe blades 117 moving, the microbots 200 a-n are controlled to move withthe blades 117. As illustrated in FIG. 4C, in embodiments each of themicrobots 200 a-n is controlled to display a respective portion of thevideo content, such that the plurality of microbots 200 a-n mirrors aportion of the video content that is covered by the microbots 200 a-n onthe display 105

Referring to FIG. 3, an exemplary microbot 200 representing a respectiveone of the plural microbots 200 a-n is shown. As used herein, microbotrefers to a miniature mobile robot with characteristic dimensions lessthan 1 mm. The term can also be used for robots capable of handlingmicrometer size components. In embodiments, the microbot 200 includes achassis 205, a controller 210, a battery 215, a motor 220, one or moresensors 225, a communication system 230, one or more propulsive elements235, and a display 240. The battery 215 stores and provides electricalpower to elements of the microbot 200, and is preferably a rechargeablebattery. The controller 210 comprises logic circuitry and/or a computerprocessor that is operatively connected to, and configured to providecontrol functions for, each of the motor 220, the one or more sensors225, the communication system 230, and the display 240.

In aspects, the communication system 230 is a system that provides forcommunication with the communication system 120 of the computer device100. For example, the communication system 230 may comprise at least oneantenna that is configured to receive wireless communication signalsfrom an antenna of the computer device 100. In this manner, the microbot200 may receive control signals from the control module 115, where thecontrol signals instruct the microbot 200 where to move on the surfaceof the display 105. In additional embodiments, the communication system230 is also configured to provide wireless communication betweenrespective ones of the microbots 200 a-n. In this manner the microbots200 a-n may communicate with one another for swarm control (e.g.,self-assembly) of the plurality of microbots 200 a-n.

The motor 220 and the one or more propulsive elements 235 cooperate toprovide a motive force that moves the microbot 200 relative to thesurface of the display 105. In embodiments, the one or more propulsiveelements 235 comprise one or more legs, wheels or other propulsiveelements that are used to propel a microbot along a surface. Inembodiments, the motor 220 is an electric motor (e.g., a direct current(DC) motor) that actuates (e.g., causes movement of) the one or more oneor more propulsive elements 235. In aspects, the one or more propulsiveelements 235 are designed to be in physical contact with a surface ofthe display 105 to support the microbot 200 over the display 105 and tomove the microbot 200 relative to the surface of the display 105.

In additional embodiments, the microbot 200 is a flying microbot such asa micro-scale flying drone. In this embodiment, the one or morepropulsive elements 235 comprise one or more rotors, propellers, or fansthat are configured to cause the microbot 200 to fly relative to thedisplay 105.

In embodiments, the one or more sensors 225 include sensors that providedata for controlling the movement of the microbot 200 on the surface ofthe display 105. The one or more sensors 225 may include a gyroscope,such as a vibrating structure gyroscope, such as amicroelectromechanical systems (MEMS) gyroscope, that provides data thatis used in directional control of the microbot 200 on the display 105.The one or more sensors 225 may also include one or more proximitysensors, such as infrared proximity sensors, that provide data that isused in determining spatial proximity of one microbot 200 relative toanother microbot (or microbots) on the display 105.

In additional embodiments, the microbot 200 includes a wire 237extending from the chassis 205 to a point of contact with the display105. In this embodiment, the outer surface of the display 105 isprovided with plural electrically conductive portions. In thisembodiment, rather than using wireless communication, the communicationsystem 120 may provide communication signals to the microbot 200 via thewire 237 when the wire 237 comes into contact with one of theelectrically conductive portions. Additionally or alternatively in thisembodiment, when the wire 237 comes into contact with one of theelectrically conductive portions, the microbot 200 transmits datadefining its identity to the communication system 120 via the wire 237,and the communication system 120 passes this identity data and alocation of the respective electrically conductive portion to thecontrol module 115, which uses this location information of therespective microbot 200 relative to the display 105 in determiningmovement schemes and control signals for this one microbot 200 and/orother ones of the plurality of microbots 200 a-n.

In additional embodiments, the microbot 200 includes a suspensionelement 239 connected between the chassis 205 and each of the one ormore propulsive elements 235. The suspension element permits relativemovement between the chassis 205 and the one or more propulsive elements235, e.g., similar to a suspension system of an automobile. In thisembodiment, the sensors 225 include one or more sensors that detect ameasure of the deflection of each respective suspension element 239. Bymeasuring the deflection of the suspension elements, the system maydetermine that a user (e.g., a person viewing the content on thedisplay) is pressing down on the microbot 200, and this information maybe used in embodiments to control aspects of the movement of one or moreof the microbots 200 a-n.

Still referring to FIG. 3, according to aspects of the invention, thedisplay 240 is a video display, such as an OLED display, that isconfigured to display a portion of the video content that is displayedon the display 105. According to aspects of the invention, while thedisplay 105 is displaying the video content, the analysis module 110(shown in FIG. 2) determines a portion of an image of the video contentthat is to be covered by a respective microbot 200 and passes thisinformation to the control module 115. Based on this, the control module115 sends a control signal to the respective microbot 200 (via thecommunication system 120) to cause the microbot 200 to: (i) move to aposition on the display 105 to coincide with a location of thedetermined portion of the image, and (ii) display the determined portionof the image on its display 240. In embodiments, the determined portionof the image is a relatively small area of the overall image that ispart of the video content being displayed on the display 105, and eachrespective one of the microbots 200 a-n moves to a different determinedportion of the image and displays that determined portion of the imageon its display 240. For example, FIG. 4B shows the microbots 200 a-nmoved to locations over determined portions of the image, but withoutdisplaying any portion of the image via their respective displays 240.FIG. 4C shows the microbots 200 a-n moved to locations over determinedportions of the image with each one of the microbots 200 a-n displaying,on its own display 240, the portion of the image that the respectivemicrobot is covering on the display 105. In this manner, the portions ofthe image that are covered by the microbots 200 a-n are reproduced onthe displays 240 of the microbots 200 a-n.

In embodiments, the portions of the image that are covered by themicrobots 200 a-n are defined by an object that is moving in the videocontent that is being displayed on the display 105, e.g., the windmillblades 117 in the example shown in FIGS. 4A-C. In aspects, the videocontent comprises plural image frames that show movement of the objectwhen displayed in succession, e.g., the windmill blades 117 rotate inthe direction D as the video content plays. Accordingly, as the videocontent plays on the display 105 and the displayed image changes fromone image frame to the next, the microbots 200 a-n move relative to thedisplay 105 to remain in synchronous motion with the object andsimultaneously display their respective portions of the image.

According to aspects of the invention, the analysis module 110determines the portions of the image that are to be covered by themicrobots 200 a-n by analyzing the video content that is being displayedon the display 105. In an embodiment, the video content includes datathat defines the portions of each image frame to be covered by themicrobots 200 a-n. For example, each frame of the video content mayinclude data (e.g., metadata) that defines a location of a portion ofthe image (e.g., an x-y coordinate relative to the display 105). Inembodiments, the analysis module 110 obtains (e.g., extracts or decodes)and analyzes this data from the video content and passes it to thecontrol module 115, which controls one of the microbots 200 a-n to moveto the location defined in the data. Each frame of the video content mayhave data that defines respective locations of plural such portions,such that the analysis module 110 and the control module 115 cooperateto cause plural ones of the microbots 200 a-n to move to the pluralrespective locations defined in the data. Each frame of the videocontent may also have data that defines a portion of the image thatcorresponds to the defined location, in which case the analysis module110 obtains this data from the video content and passes it to thecontrol module 115, which controls one of the microbots 200 a-n todisplay on its display 240 the portion of the image defined by the data.

In another embodiment, the analysis module 110 determines the portionsof the image that are to be covered by the microbots 200 a-n using realtime video analysis techniques. In this embodiment, the analysis module110 is configured to perform real time object detection in the series ofimage frames that make up the video content. The real time objectdetection may be based on computer vision techniques, for example, andmay include pre-processing steps such as image filtering. In thismanner, the analysis module 110 may be programmed to detect the locationof a moving object in the video content. In this embodiment, afterdetecting a moving object in the video content, the analysis module 110divides the object into discrete portions, determines a location of eachdiscrete portion relative to the display 105, and cooperates with thecontrol module 115 to cause the microbots 200 a-n to move to locationson the display 105 corresponding to the determined portions of thedetected object. In this embodiment and based on the dividing thedetected object into the discrete portions, the analysis module 110 mayalso pass video data for each discrete portion to the control module115, which controls each of the respective microbots 200 a-n to displayon its display 240 the video data for its discrete portion. In aspects,the analysis module 110 and control module 115 analyze the video contentand control the microbots 200 a-n in this manner for each successiveframe of the video content.

In the embodiment that is based on real time object detection, it isconceivable that the analysis module 110 may detect more than one movingobject in the video content. Accordingly, in response to the analysismodule 110 detecting plural moving objects in the video content, theanalysis module 110 may be configured to select a single object to fromthe plural detected objects. In aspects, the selection is based onpredefined criteria including at least one selected from the groupconsisting of: size of each detected object (e.g., stronger preferencefor selecting the largest sized object from the plural detectedobjects); location of each detected object in the image frame (e.g.,stronger preference for selecting objects that are closer to the centerof the image frame); focus of each object (e.g., stronger preference forselecting objects that are in focus); and contrast of the object with abackground of the image (e.g., stronger preference for selecting objectswith higher contrast).

According to an embodiment of the invention, the analysis module 110 andthe control module 115 are configured to control all of the microbots200 a-n individually. In this mode, the analysis module 110 determines alocation for each respective one of the microbots 200 a-n (e.g., usingthe techniques already described) and passes the determined locations tothe control module 115. The control module 115 then sends respectivecontrol signals to each one of the microbots 200 a-n, e.g., via thecommunication system 120, that cause each one of the microbots 200 a-nto move to its determined location on the display 105. In aspects, thecontrol signal transmitted to a respective one of the microbots 200 a-nincludes a direction of movement and a speed for the respective each oneof the microbots 200 a-n. This is performed for each frame of the videocontent.

According to another embodiment of the invention, the analysis module110 and the control module 115 are configured to provide an objective tothe microbots 200 a-n collectively, and the microbots 200 a-n areconfigured to use self-assembly techniques to move to respectivelocations on the display 105 to achieve the objective. In thisembodiment, the analysis module 110 detects a moving object in themanner described here and determines locations that define an outline(e.g., a perimeter) of the detected object; however the analysis module110 does not divide the detected object into discrete portions. Theanalysis module 110 passes the data defining the locations of theoutline of the detected object to the control module 115, whichformulates a command to the microbots 200 a-n as a group to achieve theobject of self-assembling in a manner to cover an area on the display105 that is bounded by the outline. For example, in the use caseillustrated in FIGS. 4A-C, the analysis module 110 would determinelocations that define an outline of the windmill blades 117. Based onthis, the control module 115 would instruct the microbots 200 a-n toself-assemble in an area on the display bounded by the locations thatdefine the outline of the windmill blades 117. In response, themicrobots 200 a-n would communicate with one another and utilizeself-assembly techniques to achieve the instructed objective.

According to aspects of the invention, the analysis module 110 isconfigured to decide whether or not to control the microbots 200 a-n tomove with an object in the video content based on one or moreparameters. For example, the analysis module 110 may be programmed witha maximum movement speed of the microbots 200 a-n and may determine thatthe detected object is moving faster than the maximum movement speed ofthe microbots 200 a-n. Based on this determination, the analysis module110 decides to not attempt to control the microbots 200 a-n to move withthe detected object. In another example, the analysis module 110 may beprogrammed with a physical size of each of the microbots 200 a-n, andmay decide to not deploy the microbots based on determining that thedetected object is too small relative to the size of an individualmicrobot or that the detected object is too large relative to thecollective size of all the microbots 200 a-n. In another example, theanalysis module 110 may be programmed to cause the microbots to moveonly within a predefined area of the display 105, e.g., to avoid causingthe microbots 200 a-n to move too close to an edge of the display 105and risk falling off the display 105. Other parameters that may be takeninto account by the analysis module 110 in deciding whether to deploythe microbots 200 a-n include but are not limited to: boundary;dimension; size; color; location; and relative distance to surroundingobjects.

In additional embodiments, the learning module 125 (shown in FIG. 2) isconfigured to train the system based on user interaction andconfiguration pattern, feedback, and trend analysis. In this embodiment,the learning module 125 obtains user feedback indicating a user'sopinion of how well the microbots 200 a-n simulated a moving object inthe video content displayed on the display 105. In the event that thefeedback is negative, the learning module 125 instructs the analysismodule 110 to not control the microbots 200 a-n to move with this sametype of object in future video content that is displayed for this user.In the example of FIGS. 4A-C, in response to the user indicating viafeedback that the microbots 200 a-n did a very poor job of simulatingthe movement of the windmill blades 117, the learning module 125instructs the analysis module 110 to not attempt to simulate themovement of windmill blades in other video content that is displayed forthis user.

In embodiments, the learning is based on feedback from plural usersregarding a single type of object. For example, the learning module 125of plural different computer devices may be configured to obtainfeedback from plural different users. In the event that the collectivefeedback from plural different users is negative for simulating aparticular object (e.g., a windmill blade), the learning module 125instructs the analysis module 110 to not control the microbots 200 a-nto move with this same type of object in future video content that isdisplayed for any user.

FIGS. 5 and 6 show exemplary implementations of the microbot 200 on thedisplay 105 in accordance with aspects of the invention. As shown inFIG. 5, the microbot 200 physically contacts an upper surface of thedisplay 105 and moves in directions parallel to the upper surface of thedisplay 105. As shown in FIG. 6, the computer device may include aflexible membrane 600 such that the microbots 200 a-n (represented hereby individual microbot 200) are sandwiched between the upper surface ofthe display 105 and the membrane 600. In embodiments, the membrane 600is transparent or translucent, and is flexible such that it deformswhere it comes into physical contact with the microbot 200. Inembodiments, the membrane 600 is a touchscreen material, such as aresistive or capacitive touch screen material, that is operativelyconnected to the computer device 100. In this manner, an individual mayuser touchscreen inputs (e.g., swipe, pinch, etc.) to provide inputsthat control the video content that is displayed on the display 105.

In some embodiments, and as described herein, each microbot 200 includesa suspension element 239 and a sensor 225 that measures a deflection ofthe suspension element 239. According to aspects of the invention, thecontrol module 115 determines that a user is pressing down on themicrobot 200 based on the measured deflection of the suspension element239. For example, as shown in FIG. 4C, a user's hand 400 may come intophysical contact with one or more of the microbots 200 a-n. In responseto this determination, the control module 115 adjusts its control of amicrobot 200 that is being pressed by a user. For example, the controlmodule 115 may instruct the microbot 200 to increase the output of themotor 220 to attempt to overcome the force being exerted on the microbot200 by the user. Additionally or alternatively, the control module 115may instruct the microbot 200 to move faster (e.g., faster the movingobject in the video content) in order for the microbot 200 to catch upto the location of the object in the video content. This control isuseful when a user touches one or more of the microbots 200 a-n for anyreason. This control can be utilized in embodiments with the membrane600 and also in embodiments without the membrane 600.

FIG. 7 shows a flowchart of an exemplary method in accordance withaspects of the present invention. Steps of the method may be carried outin the environment of FIG. 2 and are described with reference toelements depicted in FIG. 2.

At step 705, the system displays video content on a computer display. Inembodiments, and as described with respect to FIG. 2, the computerdevice 100 displays video content on the display 105.

At step 710, the system determines a moving object in the video content(i.e., the video content of step 705). In one embodiment, and asdescribed with respect to FIG. 2, the analysis module 110 determines themoving object based on data included in or with the video content. Inthis embodiment, the video content may be tagged with data (e.g.,metadata) that defines the location of the moving object in each imageframe, and the analysis module 110 determines the object by decoding orextracting this data.

In another embodiment, and as described with respect to FIG. 2, step 710comprises the analysis module 110 determining the moving object basedusing real time image analysis such as computer vision based objectdetection. In this embodiment, the analysis module 110 is programmed touse real time object detection techniques to identify moving objects insuccessive image frames of the video content. In this embodiment, step710 may include the analysis module determining plural moving objectsand selecting a subset of the plural moving objects (e.g., one) based oncriteria, as described with respect to FIG. 2.

At step 715, the system controls microbots to move, on the display, tolocations over the object (from step 710). In one embodiment, and asdescribed with respect to FIGS. 2 and 4B, the control module 115generates and sends control signals (via the communication system 120)to each microbot individually. In this embodiment, the control signal toeach respective microbot may include data that instructs the microbot tomove to a defined location on the display 105. The control signal toeach respective microbot may also include data that defines a speed atwhich the microbot should move to the defined location on the display105 and/or a path the microbot should take in moving to the definedlocation on the display 105.

In another embodiment, and as described with respect to FIG. 2, step 715comprises the control module sending data defining an objective to atleast one of the microbots 200 a-n. In this embodiment, the microbots200 a-n then use self-assembly techniques to control their own movementto achieve the objective. The objective may include, for example, datathat defines locations of an outline of the object on the display 105.In this example, the microbots 200 a-n communicate with one another tocontrol their own movement to move to locations on the display 105 thatare within an area defined by the outline.

At step 720, the system controls the microbots to display portions ofthe image of the object (from step 710). In embodiments, and asdescribed with respect to FIGS. 2 and 4C, the analysis module 110determines respective portions of the image for each of the microbots200 a-n to display on its display screen 240, and the control module 115generates and sends control signals to the microbots to cause themicrobots to display the determined respective portions of the image. Inembodiments, each microbot displays a portion of the object that themicrobot is covering on the display 105, e.g., as shown in FIG. 4C.

At step 725, the system detects a user touching one of the microbots andadjusts control of the microbot. In embodiments, and as described withrespect to FIG. 2, the control module 115 detects that a user istouching one of the microbots 200 a-n based on sensor data that measuresdeflection of a suspension element in the one of the microbots. Inaspects, based on this detecting the control module 115 sends anadjusted control signal to the microbot that is being touched by theuser. The adjusted control signal may comprise, for example, a change inpower, speed, or direction of the microbot.

After any of steps 715, 720, and 725, the method returns to step 705 tostart the next iteration, i.e., to display the next image frame of thevideo content. In this manner, the system causes the microbots 200 a-nto move with the moving object in the video content as the video contentis being displayed no the display 105. Steps 720 and 725 are optionaland may or may not be performed during each iteration of the method.

At step 730, the system obtains user feedback and trains based on thefeedback. In embodiments, and as described with respect to FIG. 2, thelearning module 125 obtains user feedback and provides instructions tothe analysis module 110 based on the user feedback.

In embodiments, a service provider could offer to perform the processesdescribed herein. In this case, the service provider can create,maintain, deploy, support, etc., the computer infrastructure thatperforms the process steps of the invention for one or more customers.These customers may be, for example, any business that uses technology.In return, the service provider can receive payment from the customer(s)under a subscription and/or fee agreement and/or the service providercan receive payment from the sale of advertising content to one or morethird parties.

In still additional embodiments, the invention provides acomputer-implemented method, via a network. In this case, a computerinfrastructure, such as computer system 12 (FIG. 1), can be provided andone or more systems for performing the processes of the invention can beobtained (e.g., created, purchased, used, modified, etc.) and deployedto the computer infrastructure. To this extent, the deployment of asystem can comprise one or more of: (1) installing program code on acomputing device, such as computer system 12 (as shown in FIG. 1), froma computer-readable medium; (2) adding one or more computing devices tothe computer infrastructure; and (3) incorporating and/or modifying oneor more existing systems of the computer infrastructure to enable thecomputer infrastructure to perform the processes of the invention.

The descriptions of the various embodiments of the present inventionhave been presented for purposes of illustration, but are not intendedto be exhaustive or limited to the embodiments disclosed. Manymodifications and variations will be apparent to those of ordinary skillin the art without departing from the scope and spirit of the describedembodiments. The terminology used herein was chosen to best explain theprinciples of the embodiments, the practical application or technicalimprovement over technologies found in the marketplace, or to enableothers of ordinary skill in the art to understand the embodimentsdisclosed herein.

What is claimed is:
 1. A method, comprising: displaying, by a computerdevice, video content on a display of the computer device; determining,by the computer device, a moving object in the video content; andcontrolling, by the computer device, a plurality of microbots to move,on the display, over locations of the object in the video content. 2.The method of claim 1, further comprising controlling each of theplurality of microbots to display a portion of the object.
 3. The methodof claim 1, wherein: the video content includes metadata that definesthe moving object; and the determining the moving object comprisesobtaining the metadata.
 4. The method of claim 1, wherein thedetermining the moving object comprises analyzing the video contentusing computer vision based object detection.
 5. The method of claim 4,wherein the determining the moving object comprises: determining pluralmoving objects in the video content; and selecting the moving objectfrom the plural moving objects based on predefined criteria.
 6. Themethod of claim 1, wherein the controlling the plurality of microbots tomove comprises transmitting a control signal to each one of theplurality of microbots.
 7. The method of claim 6, wherein the controlsignal transmitted to a respective one of the plurality of microbotsincludes a direction of movement and a speed for the respective one ofthe plurality of microbots to move relative to the display.
 8. Themethod of claim 1, wherein: the controlling the plurality of microbotsto move comprises transmitting a control signal to at least one of theplurality of microbots; the control signal includes data that defines anarea on the display; and the control signal includes an instruction thatcauses the plurality of microbots to self-assemble on the defined area.9. The method of claim 1, further comprising: detecting a user touchingone of the plurality of microbots; and adjusting control of the one ofthe plurality of microbots based on the detecting.
 10. The method ofclaim 1, further comprising: obtaining user feedback; and training amodule that performs the determining the moving object based on thefeedback.
 11. A computer program product, the computer program productcomprising a computer readable storage medium having programinstructions embodied therewith, the program instructions executable bya computing device to cause the computer device to: display videocontent on a display of the computer device; determine a moving objectin the video content; and control a plurality of microbots to move, onthe display, over locations of the object in the video content.
 12. Thecomputer program product of claim 11, wherein the program instructionscause the computer device to control each of the plurality of microbotsto display a portion of the object.
 13. The computer program product ofclaim 11, wherein: the video content includes metadata that defines themoving object; and the determining the moving object comprises obtainingthe metadata.
 14. The computer program product of claim 11, wherein thedetermining the moving object comprises analyzing the video contentusing computer vision based object detection.
 15. The computer programproduct of claim 11, wherein the controlling the plurality of microbotsto move comprises transmitting a control signal to each one of theplurality of microbots.
 16. The computer program product of claim 11,wherein: the controlling the plurality of microbots to move comprisestransmitting a control signal to at least one of the plurality ofmicrobots; the control signal includes data that defines an area on thedisplay; and the control signal includes an instruction that causes theplurality of microbots to self-assemble on the defined area.
 17. Thecomputer program product of claim 11, wherein the program instructionscause the computer device to: detect a user touching one of theplurality of microbots; and adjust control of the one of the pluralityof microbots based on the detecting.
 18. The computer program product ofclaim 11, wherein the program instructions cause the computer device to:obtain user feedback; and train a module that performs the determiningthe moving object based on the feedback.
 19. A system comprising: aprocessor, a computer readable memory, and a computer readable storagemedium; program instructions to display video content on a display ofthe computer device; program instructions to determine a moving objectin the video content; and program instructions to control a plurality ofmicrobots to move, on the display, over locations of the object in thevideo content such that the plurality of microbots move in synchronousmotion with the object as the video content is displayed on the display,wherein the program instructions are stored on the computer readablestorage medium for execution by the processor via the computer readablememory.
 20. The system of claim 19, further comprising programinstructions to control each of the plurality of microbots to display aportion of the object.